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(54) IMAGE COMPOSITE METHOD AND IMAGE COMPOSITE DEVICE 
(57)Abstract: 

PURPOSE: To realize a image composite method and a 
device in which a panoramic image with an optional aspect 
ratio is obtained with less deterioration in image quality. 
CONSTITUTION: Plural image pickup systems 10R, 10L are 
used to pick up an image of a common object 1 while 
overlapping part of a picture field and the plural image 
signals obtained by the image pickup system 10R, 10L are 
synthesized to form one image signal in this image ^ 
composite method. Distance information of the object 1 is 
used to conduct coordinate transformation processing and 
the image signals outputted from the image pickup systems 
10R, 10L are interpolated depending on the transformation 
coordinate and one of the interpolated image signals is 
selected with respect to an area where a remote distance 

part of the object 1 is hidden by a near distance part of the object among the overlapped area of 
the composite image signal to obtain one image signal which is outputted in the state defined at 
an optional object distance and image forming magnification from the image pickup system where 
the plural image signals outputted from the image pickup systems 10R, 10L are defined by the 
positional deviation of the visual point from the image pickup systems 10R, 10L and the angle of 
convergence of the optical axes LR, LL with respect to the visual point position and the directions 
of the optical axes LR, LL. 
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l, 4 l*<Dtiit}{m<D¥-i%mi5<i:Zf&-7* — 
-£*4 2i, 4 2R©U*fi^©¥J3ffi*#i&;fcat> V^f 
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* NOTICES * 

Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may 

not reflect the original precisely. 

2 **** shows the word which can not be translated. 

3. In the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim l] In an image composition method which a part of **** is made to 
overlap using two or more image pick-up systems, picturizes a common 
photographic subject, compounds two or more picture signals acquired by 
each image pick-up system, and creates one picture signal Perform 
coordinate transformation processing using distance information on a 
photographic subject, and a picture signal outputted from said each image 
pick-up system according to this conversion coordinate is interpolated. A 
long distance portion of a photographic subject is related with a field hidden 
in a short-distance body among overlap fields of a synthetic picture signal. 
The object distance of an image pick-up system as which the direction of a 
view location and an optical axis is defined by the amount of location gaps of 
a view from said each image pick-up system, and angle of convergence of an 
optical axis in two or more picture signals outputted from said each image 
pick-up system by choosing one of interpolated picture signals to arbitration, 
An image composition method characterized by performing image 
composition so that it may become one picture signal outputted in the 
condition of defining as an image formation scale factor. 

[Claim 2] An image composition method characterized by being picturized by 
image pick-up system whose field by which a long distance portion of a 
photographic subject is this **(ed) among overlap fields of a synthetic picture 
signal in an image composition method according to claim 1 about a field 
hidden in a short- distance body decreases most, and choosing an 
interpolated picture signal. 

[Claim 3] In an image synthesizer unit which a part of **** is made to 
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overlap using two or more image pick-up systems, compounds a picture 
signal which picturized a common photographic subject and was acquired, 
and creates one picture signal Two or more image memories which memorize 
image data picturized by said each image pick-up system, respectively, The 
corresponding-points extract section which detects a coordinate 
corresponding to predetermined image data, and is memorized as 
corresponding-points information about image data remembered to be the 
parameter storage section which memorizes optical information about said 
each image pick-up system in each image memory, respectively, The distance 
information calculation section which computes distance information from 
corresponding-points information memorized by optical information 
memorized by said parameter storage section and said corresponding-points 
extract section, A 2-dimensional coordinate generated in the address 
generation section which carries out sequential generating and outputs a 
2-dimensional coordinate of a display image, and said address generation 
section The coordinate transformation section made into a 2-dimensional 
coordinate of image data which changes from optical information memorized 
by said parameter storage section and distance information computed in said 
distance information calculation section, and is memorized in said two or 
more image memories, respectively, The image data-interpolation section 
which performs interpolation processing to image data memorized in each 
image memory by 2-dimensional coordinate after being changed in said 
coordinate transformation section, Making two or more image data 
interpolated in said image data-interpolation section compound as one 
picture signal Inside of a field which an image pick-up image by each image 
pick-up system overlaps, About a field where a long distance portion of a 
photographic subject which corresponding-points information memorized by 
said corresponding-points extract section shows is hidden in a short- distance 
body, one of output signals of said image data-interpolation section is chosen. 
An image synthesizer unit characterized by having the image convolution 
transformation processing section possessing the image composition section 
which performs image composition. 

[Claim 4] It is the image synthesizer unit which the image convolution 
transformation processing section is picturized in an image synthesizer unit 
according to claim 3 by image pick-up system whose field by which a long 
distance portion of a photographic subject is this **(ed) among overlap fields 
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of a synthetic picture signal about a field hidden in a short-distance body 
decreases most, and is characterized by choosing an interpolated picture 
signal. 

[Claim 5] It is the image synthesizer unit according to claim 3 which is 
equipped with the following and characterized by for the image convolution 
transformation processing section being picturized by image pick-up system 
with many small fields which a long distance portion of a photographic 
subject correlates more in said image correlation section among overlap 
fields of a synthetic picture signal about a field hidden in a short- distance 
body, and choosing an interpolated picture signal. The 2nd formed image 
memory which memorizes image data interpolated by the image 
data-interpolation section The image correlation section which asks a small 
field for similarity of each smallness field as cutoff, and performs each image 
data a long distance portion of a photographic subject of the fields which an 
image pick-up image by each image pick-up system overlaps is remembered 
to be in decision of a field hidden in a short* distance body in said 2nd image 
memory by whether this decision value is lower than a predetermined value 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Industrial Application] This invention relates to the compound eye image 
pick-up equipment which compounds two or more images and offers the 
panorama image of the aspect ratio of arbitration especially about compound 
eye image pick-up equipment. 
[0002] 

[Description of the Prior Art] It is used, in case the aspect ratio conversion 
method which trims the upper and lower sides or right and left of a screen is 
learned as an aspect ratio conversion method which changes the aspect ratio 
of a television screen at the time of an output, for example, the aspect ratio (4 
to 3) of the television screen of NTSC system is conventionally changed into 
the aspect ratio (16 to 9) of the television screen of HD method or ED2 
method. 

[0003] Although the problem on image quality is not produced when an 
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aspect ratio picturizes a photographic subject with the NTSC camera using 
the image sensors of 4 to 3 and an aspect ratio outputs the image to the 
monitor for NTSC of 4 to 3 especially since these aspect ratio conversion 
methods are methods using some picturized images, there is a trouble that 
horizontal angles of view decrease about by 1/3. 

[0004] The compound eye image pick-up equipment which obtains one high 
definition image is proposed by the Institute of Image Electronics Engineers 
of Japan draft 90-03-04, p23-28, etc. by on the other hand compounding the 
overlap field of two or more images which picturize a common photographic 
subject and are obtained using two or more image pick-up systems. Moreover, 
the compound eye image pick-up equipment which outputs the large image 
of **** is proposed by JP, 5- 110926, A etc. by compounding each image which 
inputted two or more images and performed coordinate transformation 
processing. 

[0005] By applying such compound eye image pick-up equipment and 
compound eye image pick-up equipment, a part of **** can be made to be 
able to overlap using two or more image pick-up systems, two or more images 
which picturize a common photographic subject and are obtained can be 
compounded, and the compound eye image pick-up equipment which creates 
one image of the aspect ratio of arbitration can be constituted. 
[0006] Namely, two image pick-up systems (left-hand side image pick-up 
system 110L and right-hand side image pick-up system 110R) are used like 
the compound eye image pick-up equipment shown in drawing 7 . By 
compounding the left-hand side image PL obtained by left-hand side image 
pick-up system 110L, and the right-hand side image PR obtained by 
right-hand side image pick-up system 110R with a central processing unit 
(CPU) 120, after making a part of **** overlap and picturizing the common 
photographic subject 101 One image PL+R of the aspect ratio of arbitration 
can be created. 

[0007] However, the method of obtaining the image of a predetermined 
aspect ratio also in the above-mentioned conventional example is not shown, 
but if the method which trims the image after composition is applied, image 
quality -deterioration will produce it. 

[0008] In the equipment to which these people use two or more image 
pick-up systems for, and overlap and picturize a part of **** with the 
"compound eye image pick-up equipment" of Japanese Patent Application No 
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No. 223544 [ five to ] in order to solve an above-mentioned trouble The object 
distance of the image pick-up system as which the direction of a view 
location and an optical axis is defined by the amount of location gaps of the 
view from each above-mentioned image pick-up system, and the angle of 
convergence of an optical axis in two or more picture signals outputted from 
each above-mentioned image pick-up system to arbitration, The image 
pick-up equipment which carries out convolution transformation was offered 
so that it might become one picture signal outputted in the condition of 
defining as an image formation scale factor, there was little image quality 
deterioration and, moreover, how to obtain the image with which distortion 
produced on the occasion of congestion was amended was shown. 
[0009] Hereafter, the above-mentioned conventional example is explained 
briefly. Drawing 8 is constituted by 2 sets of image pick-up systems, 
right-hand side image pick-up system 81 OR which is drawing showing basic 
arrangement of the above-mentioned conventional example, and is the 1st 
and 2nd image pick-up optical system, and left-hand side image pick-up 
system 810L. 

[0010] 801 are a photographic subject side which is a plane common to each 
image pick-up system among drawing. Left-hand side image pick-up 
optical-system 811L and right-hand side image pick-up optical-system 811R 
are image pick-up optical system which has equivalent specification, and, 
generally a zoom lens is used. 812L and 812R are the right-hand side and 
left-hand side image sensors which have equivalent specification similarly, 
and solid state image sensors, such as the camera tubes, such as Saticon, or 
CCD, are used. These opticals axis LL and an optical axis LR are arranged in 
the condition that theta inclined in symmetry to normal 00 1 of the 
photographic subject side 801 so that the conditions which the specified 
quantity of each image pick-up **** overlaps according to the screen of the 
selected aspect ratio may be fulfilled (however, the point O point on the 
photographic subject side 801). In addition, 2theta is called an angle of 
convergence, theta Moreover, each of 802L and 802R is a body side [ **** ], 
and leans to each image sensors 812L and 812R to the photographic subject 
plane 1, respectively. Points OL and OR are intersections of opticals axis LL 
and LR and the photographic subject side 801, respectively, and Points CL 
and CR are the principal points (in detail principal point by the side of a 
photographic subject) of each image pick-up optical system 81 1L and 811R. 
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There are a variable power group and a focus group in each image pick-up 
optical system 811L and 811R, and the encoder for acquiring the positional 
information of the drive system which drives these, and the direction of an 
optical axis is formed. Moreover, the encoder which detects the device system 
which rotates an image pick-up system in a plane including an optical axis, a 
drive system, and an angle of rotation is formed in each image pick-up 
system. And an angle -of- convergence control system sets up the 
control-objectives value of the angle of convergence according to the output 
signal of each encoder so that the image of a predetermined aspect ratio may 
be obtained, and the congestion control is performed. 

[0011] The decision method of the control-objectives value of an angle of 
convergence is explained using drawing 8 and drawing 9 below. 
[0012] The image formation scale factor of left-hand side image pick-up 
optical-system 811L shown in drawing 8 and right-hand side image pick-up 
optical-system 81 1R is set to beta, the object distance (distance between 
Point OL and Point CL and distance between Point OR and Point CR) is set 
to z, and distance (base length) of Point CL and Point CR is set to 2d. this 
time -- distance [ from the photographic subject plane 1 ] z' ■■ Point O 
'normal OO left to the side 1 - so that a view may be taken at the upper point 
and the imagination image formation scale factor in that view may become 
beta* When the imagination image surface (that is, the distance of a view and 
the image surface beta'z 1 ) is taken, virtual image surface IL+R by which the 
image surface IL of left-hand side image-sensors 812L and the image surface 
IR of right-hand side image-sensors 812R were compounded is shown in 
drawing 9 . 

[0013] drawing 9 setting - a point -- AL -- a point - BL a point - CL - 
and - DL - respectively -- left-hand side image sensors - 12 -- L -- the 
image surface - IL - a vertical angle - a top - a point being shown -- 
imagination - the image surface " IL -- + -- R -- a top " a point -- AL ■- ' - a 
point -- BL -- 1 -- a point -- CL - 1 - and DL 1 corresponding . moreover - 
a point AR -- a point -- BR - a point - CR - and - DR - respectively -- 
right-hand side " image sensors -- 812 -- R -- the image surface - IR - a 
vertical angle ■■ a top " a point being shown -■ imagination - the image 
surface -- IL+R - a top - a point -- AR - 1 ~ a point -- BR - 1 - a point -- CR " 
1 -- and -- DR " 1 - corresponding . Furthermore, Point EL and Point floor line 
show the point on the vertical side which takes the lead in the overlap on the 
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image surface IL of left-hand side image-sensors 812L, respectively, and 
Point ER and Point FR show the point on the vertical side which takes the 
lead in the overlap on the image surface IR of right-hand side image-sensors 
812R, respectively. Here, both the point EL and the point ER correspond to 
point E' on virtual image surface IL+R, and both the point floor line and the 
point FR correspond to point F f on virtual image surface IL+R. 
[0014] If the system of coordinates in each image surfaces IL and IR are 
defined using a x axis and an illustration perpendicular direction as the 
y-axis for the horizontal direction in drawing 9 while making the center of 
each image surfaces IL and IR into a zero, the image point on the image 
surface IR of right-hand side image-sensors 812R (xR, yR) is equivalent to 
the image point (xR\ yR f ) expressed with (l) type and (2) types on virtual 
image surface IL+R. 
[0015] 

[Equation l] 

xR - {(xRcostheta+betazsintheta+betad) /(-xRsin theta+beta zO) xbetaV 

(1) 

yR'= {yR/(-xRsin theta+beta z')} xbeta'z* (2) 

Moreover, the image point on the image surface IL of left-hand side 
image-sensors 812L (xL, yL) is equivalent to the image point (xL\ yLO 
expressed with (3) types and (4) types on virtual image surface IL+R. 
[0016] 

[Equation 2] 

xL - {(xLcostheta-betazsintheta-betad) /(xLsintheta+betaz 1 )} xbeta'z 1 (3) 

yL'= {yL/(xLsintheta+betaz')} xbeta'z 1 (4) 

Therefore, since the image on two or more image sensors which congestion 
produces by performing geometric transform processing as shown in an 
above-mentioned (l) formula - (4) type is compoundable in the image on the 
one virtual image surface, the image with which distortion produced by 
congestion was amended can be obtained by preparing the image convolution 
transformation processing section (un-illustrating) which performs such 
geometric transform processing. 

[0017] However, in the above-mentioned conventional example, in having 
the distance distribution whose photographic subject contains the body with 
which distance differs remarkably to a background, the following troubles 
arise. 
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[0018] That is, if geometric transform processing by the (l) type - (4) type is 
performed supposing a case as Body A is in the location of distance zA as 
shown in drawing 8 , only in the part of the difference of z and zA, the body 
location on the virtual image surface will shift [ the object distance ]. In this 
case, the location (xR", yR") to the image point on the image surface of the 
1st image sensors on the ideal virtual image surface for creating the 
synthetic image which a location gap does not produce (xR, yR), and the 
location (xL", yL") to the image point on the image surface of the 2nd image 
sensors (xL, yL) become like the formula shown below, respectively. 
[0019] 

[Equation 3] 

xR" ={[xRcostheta+betazsintheta+betad(z/zA) ]/[-xRsin theta+beta 

z(l-z/zA) cos theta+beta z' (z/zA)]} x beta'z 1 (5) 

yR' -{yR/[-xRsin theta+beta z(l-z/zA) cos theta+beta z' (z/zA)] 
}xbeta'z ? (6) 

xL" -- ={[xLcostheta-betazsintheta-betad(z/zA) ]/[xLsintheta+betaz(l-z/zA) 

costheta+betaz 1 (z/zA)]} x beta'z 1 - (7) 

yL"= {yL/[xLsintheta+betaz(l-z/zA) costheta+betaz f (z/zA)]} 
xbeta'z' (8) 

Therefore, under the effect of photographic subject distance, a gap arises and 

the amount of gaps becomes like a degree type, respectively. 

[0020] 

[Equation 4] 

**xR=xR"xR\ ** y R=yR"-yR f **xL=xL"-xL\ ** y L=yL"-yL' ( 9 ) 

For this reason, as shown in drawing 10 , the gap produced only **yR and ** 
yL in **xR, **xL, and the direction of y in the x directions to the ideal 
location a, and the point of the arbitration on Body A had the trouble that the 
image quality of a synthetic image deteriorated. 

[0021] Moreover, since the field shown with the slash in an overlap field in 
drawing 10 is a field in which a background is hidden by Body A, although it 
is an overlap field, it is not picturized with both 1st and 2nd image sensors. 
Therefore, the portion of IL shown in drawing 10 was picturized only as an 
image of the 1st image sensors, the portion of IR was picturized only as an 
image of the 2nd image sensors, even if it compounded the image after 
coordinate transformation, the amount of [ a body portion and ] background 
lapped, and there was a trouble that the image quality of a synthetic image 
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deteriorated. 

[0022] This invention is made in view of the trouble which a Prior art which 
was mentioned above has, there is little image quality deterioration, and it 
aims at realizing the image composition method that the panorama image 
which is the aspect ratio of arbitration is obtained, and equipment 
equipment. 
[0023] 

[Means for Solving the Problem] In an image composition method which a 
part of **** is made to overlap an image composition method of this 
invention using two or more image pick-up systems, it picturizes a common 
photographic subject, compounds two or more picture signals acquired by 
each image pick-up system, and creates one picture signal Perform 
coordinate transformation processing using distance information on a 
photographic subject, and a picture signal outputted from said each image 
pick-up system according to this conversion coordinate is interpolated. A 
long distance portion of a photographic subject is related with a field hidden 
in a short-distance body among overlap fields of a synthetic picture signal. 
The object distance of an image pick-up system as which the direction of a 
view location and an optical axis is defined by the amount of location gaps of 
a view from said each image pick-up system, and angle of convergence of an 
optical axis in two or more picture signals outputted from said each image 
pick-up system by choosing one of interpolated picture signals to arbitration, 
It is characterized by performing image composition so that it may become 
one picture signal outputted in the condition of defining as an image 
formation scale factor. 

[0024] In this case, a field by which a long distance portion of a photographic 
subject is this **(ed) among overlap fields of a synthetic picture signal about 
a field hidden in a short -distance body is picturized by image pick-up system 
which decreases most, and may choose an interpolated picture signal. 
[0025] In an image synthesizer unit which a part of **** is made to overlap 
an image synthesizer unit of this invention using two or more image pick-up 
systems, it compounds a picture signal which picturized a common 
photographic subject and was acquired, and creates one picture signal Two 
or more image memories which memorize image data picturized by said each 
image pick-up system, respectively, The corresponding-points extract section 
which detects a coordinate corresponding to predetermined image data, and 



9/25 



Japanese Publication number : 08-018857 



is memorized as corresponding-points information about image data 
remembered to be the parameter storage section which memorizes optical 
information about said each image pick-up system in each image memory, 
respectively, The distance information calculation section which computes 
distance information from corresponding-points information memorized by 
optical information memorized by said parameter storage section and said 
corresponding-points extract section, A 2- dimensional coordinate generated 
in the address generation section which carries out sequential generating 
and outputs a 2-dimensional coordinate of a display image, and said address 
generation section The coordinate transformation section made into a 
2-dimensional coordinate of image data which changes from optical 
information memorized by said parameter storage section and distance 
information computed in said distance information calculation section, and is 
memorized in said two or more image memories, respectively, The image 
data-interpolation section which performs interpolation processing to image 
data memorized in each image memory by 2-dimensional coordinate after 
being changed in said coordinate transformation section, Making two or 
more image data interpolated in said image data-interpolation section 
compound as one picture signal Inside of a field which an image pick-up 
image by each image pick-up system overlaps, It is characterized by having 
the image convolution transformation processing section possessing the 
image composition section which chooses one of output signals of said image 
data-interpolation section about a field where a long distance portion of a 
photographic subject which corresponding-points information memorized by 
said corresponding-points extract section shows is hidden in a short- distance 
body, and performs image composition. 

[0026] In this case, the image convolution transformation processing section 
is good, though a field by which a long distance portion of a photographic 
subject is this **(ed) among overlap fields of a synthetic picture signal about 
a field hidden in a short-distance body is picturized by image pick-up system 
which decreases most and chooses an interpolated picture signal. 
[0027] Moreover, the 2nd formed image memory which memorizes image 
data interpolated by the image data-interpolation section, A long distance 
portion of a photographic subject of the fields which an image pick-up image 
by each image pick-up system overlaps decision of a field hidden in a 
short-distance body A small field is asked for similarity of each smallness 
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field for each image data memorized in said 2nd image memory as cutoff. 
The image correlation section performed by whether this decision value is 
lower than a predetermined value is prepared. The image convolution 
transformation processing section It is good, though a long distance portion 
of a photographic subject is picturized by image pick-up system with many 
small fields correlated more in said image correlation section and chooses an 
interpolated picture signal among overlap fields of a synthetic picture signal 
about a field hidden in a short -distance body. 
[0028] 

[Function] It restricts no being picturized by image pick-up systems about 
the field in which the long distance portion of a photographic subject is 
hidden with a short- distance body among overlap fields, but if the image 
pick-up system by each image pick-up system is compounded, image quality 
will deteriorate. 

[0029] In the image composition method and equipment of this invention, 
since the image obtained by one image pick-up system is chosen and it 
performs image composition in performing image composition of the above 
fields, the image quality of a synthetic image does not deteriorate. 
[0030] Since most actual image pick-up images will be used when the field 
hidden in the case of image selection chooses the image pick-up system 
which decreases most, a more nearly actually near exact image is 
compounded. 

[0031] Since a mutually related high interpolation image is chosen more 
when the image correlation section is prepared, even if it is the case where 
corresponding-points extract processing is not performed completely, few 
synthetic images of distortion with which interpolation processing was 
performed smoothly are created. 
[0032] 

[Example] Next, the example of this invention is explained with reference to 
a drawing. 

[0033] Example 1 drawing 1 is drawing showing basic arrangement of the 
image pick-up system in one example of the compound eye image pick-up 
equipment of this invention. 

[0034] This example performs image composition using the same compound 
eye image pick-up equipment as what was indicated by Japanese Patent 
Application No. No. 223544 [ five to ] by these people fundamentally. The 
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photographic subject side 1 in drawing 1 , right-hand side body side 2R, 
left-hand side body side 2L, right-hand side image pick-up system 10R, 
left-hand side image pick-up system 10L, Each of right-hand side image 
pick-up optical-system 11R, left-hand side image pick-up optical- system 11L 
and right-hand side image-sensors 12R, and left-hand side image-sensors 
12L The photographic subject side 801 shown in drawing 8 , right-hand side 
body side 802R, left-hand side body side 802L, Since it is the same as that of 
right-hand side image pick-up system 810R, left-hand side image pick-up 
system 810L, right-hand side image pick-up optical-system 811R, left-hand 
side image pick-up optical-system 81 1L and right-hand side image-sensors 
812R, and left-hand side image-sensors 812L, explanation is omitted. 
[0035] The compound eye image pick-up equipment of this example contains 
two image pick-up systems, left-hand side image pick-up system 10L and 
right-hand side image pick-up system 10R. Here, left-hand side image 
pick-up system 10L consists of left-hand side image pick-up optical- system 
11L which consists of a zoom lens, and left-hand side image-sensors 12L 
which consists of solid state image sensors, such as the camera tubes, such 
as Saticon, or CCD. Moreover, right-hand side image pick-up system 10R has 
specification equivalent to left-hand side image pick-up system 10L, and 
consists of right-hand side image pick-up optical-system 11R which consists 
of a zoom lens, and right-hand side image-sensors 12R which consists of solid 
state image sensors, such as the camera tubes, such as Saticon, or CCD. 
[0036] In addition, although it is easy to be the thing of a single pipe type 
(veneer type) as shown in drawing 1 as left-hand side image-sensors 12L and 
right-hand side image-sensors 12R, you may be the thing of the two-pipe 
type (2 board type) through color- separation optical system, or a three-pipe 
type (3 board type). 

[0037] As the conventional example shown in drawing 8 of the same 
configuration also explained, left-hand side image pick-up optical-system 
11L and right-hand side image pick-up optical- system 11R are arranged 
after the angle theta has inclined in symmetry to normal OO f of the 
photographic subject plane 1, so that the optical axis LL of left-hand side 
image pick-up optical-system 11L and the optical axis LR of right-hand side 
image pick-up optical- system 11R may fulfill the conditions which the 
specified quantity of each image pick-up **** overlaps according to the 
screen of the selected aspect ratio. Therefore, as for right-hand side body side 
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2R [ **** ], only an angle theta inclines to left-hand side body side 2L [ **** / 
left-hand side image-sensors 12L ] and right-hand side image -sensors 12R to 
the photographic subject plane 1, respectively. 

[0038] Next, the configuration of left-hand side image pick-up optical-system 
11L and right-hand side image pick-up optical- system 11R is explained with 
reference to drawing 2 on the assumption that the basic arrangement 
mentioned above. Drawing 2 is the outline block diagram showing the 
configuration of the left-hand side image pick-up optical system shown in 
drawing 1 , and right-hand side image pick-up optical system. 
[0039] The lens groups 21L-24L in which left-hand side image pick-up 
optical-system 11L contains variable power group 22L and focus group 24L, 
Zoom motor 25L which is a drive system for driving variable power group 
22L, Focal motor 26L which is a drive system for driving focus group 24L, 
The device system (un-illustrating) for rotating left-hand side image pick-up 
optical-system 11L and left-hand side image-sensors 12L as one in a plane 
including an optical axis LL and a drive system (angle-of-convergence motor 
27L), and angle-of-convergence encoder 28L for detecting the angle of 
rotation of angle-of-convergence motor 27L are included. In addition, as 
angle-of-convergence encoder 28L, an external member like a rotary encoder 
may be used, for example, and what detects an angle of rotation by the drive 
method, for example for drive-system itself like a pulse motor may be used. 
[0040] Zoom motor 25of left-hand side image pick-up optical- system 11L L is 
driven by the zoom control section (un-illustrating) into which a zoom signal 
is inputted from the outside so that left-hand side image pick-up 
optical- system 11L may become a predetermined focal distance. Moreover, 
focal motor 26L is driven by the zoom control section (un-illustrating) into 
which a focal signal is inputted from the outside so that the focus of left-hand 
side image pick-up optical- system 11L may suit to a predetermined 
photographic subject distance. 

[0041] Zoom encoder 41of left-hand side image pick-up optical-system 11L L 
is for acquiring the positional information of the direction of an optical axis of 
each lens group (the direction of an optical axis LL) established in variable 
power group 22L, and can find the focal distance of left-hand side [ output 
signal / of zoom encoder 41L ] image pick-up optical-system 11L. Moreover, 
focal encoder 42L is for acquiring the positional information of the direction 
of an optical axis of each lens group (the direction of optical-axis LL) 
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established in focus group 24L, and can find the photographic subject 
distance of left-hand side [ output signal / of focal encoder 42L ] image 
pick-up optical-system 11L. In addition, as zoom encoder 41L and focal 
encoder 42L, an external member like a potentiometer may be used, for 
example, and what detects the positional information of the direction of an 
optical axis of a lens by the drive method, for example for drive-system itself 
like a pulse motor may be used. 

[0042] Right-hand side image pick-up optical-system 11R is constituted like 
left-hand side image pick-up optical-system 11L. In addition, zoom motor 
25of right-hand side image pick-up optical-system 11R R is driven by said 
zoom control section so that it may become the same focal distance as 
left-hand side image pick-up optical-system 11L. Moreover, focal motor 26of 
right-hand side image pick-up optical- system 11R R is driven by said zoom 
control section so that a focus may suit the same photographic subject 
distance as left-hand side image pick-up optical-system 11L. 
[0043] Next, the angle -of- convergence control system 50 of the compound eye 
image pick-up equipment of this example is explained with reference to 
drawing 3 . Drawing 3 is the block diagram showing the configuration of the 
angle-of-convergence control system in one example of the compound eye 
image pick-up equipment of this invention. 

[0044] The angle-of-convergence control system 50 consists of an operation 
control section 51 which determines the control-objectives value of an angle 
of convergence that the image after convolution transformation will serve as 
a predetermined aspect ratio, and an angle-of-convergence control section 52 
which performs angle-of-convergence control according to the output signal 
of the operation control section 51 according to the optical parameter of 
right-hand side image pick-up optical- system 11R and left-hand side image 
pick up optical- system 11L with which an image formation scale factor and 
the object distance are expressed at least. 

[0045] The output signal of zoom encoder 41L of left-hand side image pick-up 
optical-system 11L and the output signal of focal encoder 42L, the output 
signal of zoom encoder 41R of right-hand side image pick-up optical-system 
11R, and the output signal of focal encoder 42R and the control signal S from 
a system controller 60 are inputted into the operation control section 51. 
Moreover, the output signal of angle-of-convergence encoder 28L of left-hand 
side image pick-up optical-system 11L, the output signal of 
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angle-of-convergence encoder 28R of right-hand side image pick-up 
optical- system 11R, and the output signal of the operation control section 51 
are inputted into the angle-of-convergence control section 52. In addition, 
angle-of-convergence motor 27of angle-of-convergence motor 27L and 
right-hand side image pick-up optical-system 11R of left-hand side image 
pick up optical-system 11L R is driven with the output signal of the 
angle-of-convergence control section 52, respectively. 

[0046] If variable power group 22L drives by zoom motor 25of left-hand side 
image pick-up optical-system 11L L, the output signal of zoom encoder 41L 
which shows the positional information of the direction of an optical axis of 
each lens group established in variable power group 22L will be inputted into 
the operation control section 51. Moreover, if variable power group 22R 
drives by zoom motor 25of right-hand side image pick-up optical-system 11R 
R, the output signal of zoom encoder 41R which shows the positional 
information of the direction of an optical axis of each lens group established 
in variable power group 22R will be inputted into the operation control 
section 51. 

[0047] If focus group 24L drives by focal motor 26of left-hand side image 
pick-up optical- system 11L L, the output signal of focal encoder 42L which 
shows the positional information of the direction of an optical axis of each 
lens group established in focus group 24L will be inputted into the operation 
control section 51. Moreover, if focus group 24R drives by focal MUMOTA 
26of right-hand side image pick-up optical-system 11R R, the output signal 
of focal encoder 42R which shows the positional information of the direction 
of an optical axis of each lens group established in focus group 24R will be 
inputted into the operation control section 51. 

[0048] In the operation control section 51, if the control signal S which shows 
directions of optical-axis rotation of left-hand side image pick-up 
optical-system 11L and right-hand side image pick-up optical-system 11R 
was inputted from the system controller 60, after the control* objectives value 
of angle-of-convergence 2theta will be created according to the output signal 
of each zoom encoders 41L and 41R, and the output signal of each focal 
encoders 42L and 42R, the output signal which shows the created 
control-objectives value is outputted to the angle-of-convergence control 
section 52. 

[0049] In the angle-of-convergence control section 52, the output signal of the 
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operation control section 51 is compared with the output signal of each 
angle-of-convergence encoders 28L and 28R, and the output signal for 
driving each angle-of-convergence motors 27L and 27R is created, 
respectively so that the output signal of each angle-of-convergence encoders 
28L and 28R may serve as said created control-objectives value. 
[0050] A control-objectives value is determined by the same method as the 
conventional thing which explained the compound eye image pick-up 
equipment of this example mentioned above using drawing 9 . 
[0051] In addition, since the image on two or more image sensors which have 
congestion by performing geometric transform processing as shown in an 
above-mentioned (l) formula - (4) type is compoundable in the image on the 
one virtual image surface, the image with which distortion produced by 
congestion was amended can be obtained by preparing the image convolution 
transformation processing section (un-illustrating) which performs such 
geometric transform processing. 

[0052] Moreover, this image convolution-transformation processing section 
functions as the image convolution-transformation processing section which 
carries out convolution transformation of two or more picture signals 
outputted from two or more image pick-up systems so that it may become 
one picture signal outputted in the condition define as the object distance 
and the image-formation scale factor of arbitration, from the image pick-up 
system as which a view location and the direction of an optical axis are 
defined by the amount of location gaps of the view from two or more image 
pick-up systems, and the angle of convergence of an optical axis. 
[0053] If magnitude of the image surface IL of left-hand side image-sensors 
12L and the image surface IR of right-hand side image-sensors 12R is made 
into HxV (aspect ratio H/V) and magnitude of virtual image surface IL+R is 
made into H'xV (aspect ratio H7V0 By asking for the x-coordinate of point 
AR ! after conversion of Point AR and Point BL, and point BL' from the 
above-mentioned (l) formula and the above-mentioned (3) formula, length H f 
of the illustration horizontal direction of virtual image surface IL+R is 
calculable. Moreover, length V 1 of the illustration perpendicular direction of 
virtual image surface IL+R is calculable by asking for the y-coordinate of 
point BL f after conversion of Point BL and Point DL, and point DL' from the 
above-mentioned (2) formula and the above-mentioned (4) formula. Thus, the 
aspect ratio of calculated virtual image surface IL+R is expressed with (5) 
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types. 
[0054] 

[Equation 5] 

H7V'= | sin(theta+omega)/sin(omega)-2betad/H | xH/V (10) 

However, omega=tan -1 [-H/(2betaz)]: Supposing both magnitude HxV and 
2d of base lengths of the horizontal half-field angle of an image pick-up 
system therefore the image surface IL of left-hand side image-sensors 12L, 
and the image surface IR of right-hand side image-sensors 12R are fixed, the 
above-mentioned (9) formula can be written like (10) types. 
[0055] 

[Equation 6] 

Theta=f (H7, V\ beta, z) (ll) 

That is, the congestion-control angle theta which determines aspect ratio 
H7V f of virtual image surface IL+R will be decided by the image formation 
scale factor beta and the object distance z. 

[0056] So, in the operation control section 51 shown in drawing 3 , after the 
average of the output signal of each zoom encoders 41L and 41R and the 
average of the output signal of each focal encoders 42L and 42R are 
calculated, it asks using the optical parameters (a focal distance, a principal 
point gap, etc. of a lens group which constitute each image pick-up optical 
system 11L and 11R) of each image pick-up optical system 11L and 11R from 
the two averages by which the image formation scale factor beta was 
calculated. Then, the control-objectives value of an angle of convergence is 
calculated according to the image formation scale factor beta and the object 
distance z based on the above-mentioned (10) formula so that aspect ratio 
H7V' of virtual image surface IL+R may become a predetermined aspect ratio. 
The control-objectives value of the calculated angle of convergence is sent to 
the angle-of-convergence control section 52. 

[0057] In addition, although the average of the output signal of each zoom 
encoders 41L and 41R and the average of the output signal of each focal 
encoders 42L and 42R were calculated, the output signal of one of the zoom 
encoders 41L and 41R and the output signal of one of the focal encoders 42L 
and 42R may be used here. Moreover, it has the table of the 
congestion-control angle theta according to the image formation scale factor 
beta and the object distance z in the operation control section 51, the 
control-objectives value of an angle of convergence may be calculated with 
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reference to this table, the relation between the image formation scale factor 
beta and the object distance z, and the congestion-control angle theta is 
approximated to an easy polynomial, and the congestion-control angle theta 
may be calculated by memorizing the coefficient of this polynomial to the 
operation control section 51. 

[0058] Although the image of a predetermined aspect ratio is obtained by the 
above, since the largest possible field that it is not necessary to trim and is 
overlapped in this case can be taken, a high- definition image is obtained. In 
addition, as shown in drawing 4 , in case the configuration of the field turns 
into a configuration distorted somewhat from a rectangle, for example, the 
image after synthetic processing (virtual image surface IL+R) displays the 
image of an aspect ratio 16-9 on HD monitor etc., it may display a closing line 
along with an outline. 

[0059] Drawing 5 is the block diagram showing the important section 
configuration of the 1st example of the image synthesizer unit of this 
invention. Left-hand side image pick-up system 510L and right-hand side 
image pick-up system 510R are each image pick up system which showed 
basic arrangement to drawing 1 , an angle of convergence theta is controlled 
so that the image of a predetermined aspect ratio is obtained according to the 
angle -of- converge nee control system shown in drawing 3 , and a left image 
and a right image are picturized by coincidence. The other portion is the 
image convolution transformation processing section, and a configuration 
and actuation are explained about the portion below. 

[0060] an image memory 520 - L - an image memory -- 520 -- R - the left 
-- an image - and - the right - an image " digital one image data - ****** 
memorizing storage " the section - a parameter storage " the section 
521 an image pick-up a system " a parameter -- it is - the base length 
two -- d - an angle of convergence - theta image formation -- a scale 
factor -- beta - the object distance - z -- and - an output -- an image - 
imagination - an image pick-up Moreover, the small field of image memory 
520L is first stored in the corresponding-points extract section 523 as a 
template, a template is made to carry out a parallel displacement to it, 
applying location offset, template matching with the image data of image 
memory 520R is performed, and the coordinate (xR, yR) of the right image 
corresponding to the coordinate (x L, yL) of a left image is detected, distance 
-- information - calculation - the section - 524 ■■ **** - corresponding 
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points " an extract - the section - 523 - corresponding points - information 
it is -- the left - an image - and - the right - an image " a coordinate (xL, 
yL) -- (-- xR yR ■-) a parameter - storage - the section - 521 - 
memorizing - having -- **** -- the base length - two - d - an angle of 
convergence - theta - image formation ■- a scale factor - beta - the object 
distance -- z - 

[0061] The address generation section 522 carries out sequential generating 
of the 2-dimensional coordinate of the image to output, and sends it to the 
coordinate transformation section 525. The coordinate transformation 
section 525 changes the address generated in the address generation section 
522 from the parameter of an image pick-up system and the distance 
information Z which are memorized by the parameter storage section 521, 
and searches for the 2-dimensional coordinate of the left image memorized 
by image memory 520L and image memory 520R and a right image. 
[0062] Each of the image data-interpolation sections 526L and 526R 
performs interpolation processing by the 2-dimensional coordinate after 
conversion of the coordinate transformation section 525, respectively from 
the data of the left image memorized by image memory 520L and image 
memory 520R and a right image, and outputs the image data interpolated in 
the image composition section 527. In the image composition section 527, the 
result in the image data and the corresponding-points extract section 523 
which were interpolated performs image composition, and the image data 
after compounding in the image memory 528 for an output is outputted. 
With, control of actuation of the above-mentioned whole BE ** system is 
performed by the system control section which is not illustrated. 
[0063] Here, processing in the coordinate transformation section 525 is 
explained. If the coordinate within the output image side sent from the 
address generation section 526 is made into (x\ y f ), the coordinate (xL, yL) of 
a left image and a right image, and (xR, yR) can be found by the following 
(12) - (15) types with the inverse transformation (theta -> 1 theta, d->- thing 
set to d) of (5) - (8) type. 
[0064] 

[Equation 7] 

xL= {[x , costheta-beta f z t sintheta-beta , d(z7Z) Mx'sin theta+beta ! z ! (l-z7Z) cos 
theta+beta'z (z7Z)]} xbetaz (12) 

yL= {y7[x f sin theta+beta'z'(l-z7Z) cos theta+beta'z (z7Z)]} 
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xbetaz (13) 

xR= {[x'costheta+betaVsintheta+beta'd(z7Z) ]/[-x'sin theta+beta'z'(l-z7Z) cos 
theta+beta'z (z7Z)]} xbetaz (14) 

yR={y7[-x'sin theta+beta f z'(l-z7Z) cos theta+beta'z (z7Z)3 
}xbetaz (15) 

namely, - a parameter - storage - the section 521 an image pick-up - a 
system - a parameter - the base length -- two - d an angle of convergence 
- theta -- image formation - a scale factor - beta - the object distance z -- 
an output an image - imagination - image formation -- a system image 
formation ■■ a scale factor - beta 1 -- the object distance - z - 1 - distance -- 
information - calculation - the section - 524 - an output - Z - from " (-- 12 
") " - 

[0065] Next, processing of the image composition section 527 is explained. An 
output image has four kinds of fields shown in the slash section of drawing 
10 besides the field which the left image and the right image overlapped. In 
drawing 10 , the image data-interpolation value of the right image whose 
field which is outputted as synthetic image data and shown by OR is a part 
for a right flank on the outside of the overlap section and whose image 
data-interpolation value of the left image the field shown by OL is a part for 
a left flank, and is [ image ] the output of image data-interpolation section 
526L on the outside of the overlap section is the output of image 
data-interpolation section 526R is outputted as synthetic image data. 
[0066] Moreover, the image data-interpolation value of the right image 
whose field which is outputted as synthetic image data and shown by IR is a 
part for an objective right flank by the inside of the overlap section and 
whose image data-interpolation value of the left image the field shown by IL 
in drawing 10 is a part for an objective left flank, and is [ image ] the output 
of image data-interpolation section 526L by the inside of the overlap section 
is the output of image data-interpolation section 526R is outputted as 
synthetic image data. 

[0067] In the overlap section, the average value of the image 
data-interpolation value of the left image which is the output of image 
data-interpolation section 526L, and the image data-interpolation value of 
the right image which is the output of image data-interpolation section 526R 
is outputted as synthetic image data. At this time, the result of the 
corresponding-points extract section 523 is used for decision of the above 
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field. 

[0068] That is, the field where corresponding points were detected is judged 
to be the overlap section. When corresponding points are not detected, it is 
judged as fields other than the overlap section, and the amount of right flank 
of a part for a left flank is Field OR on the outside of the overlap section on 
the outside of Field OL and the overlap section. Moreover, the amount of 
[ objective ] right flank of a part for an objective left flank is Field IR by the 
inside of Field IL and the overlap section in the inside of the overlap section. 
At this time, what depends a body portion on the image data based on an 
image pick-up system with few fields in which a long distance portion is 
hidden with a short- distance body using distance information is treated as a 
body portion. By performing such image composition, most actual image 
pick-up images will be used, and a more nearly actually near exact image is 
compounded. 

[0069] Moreover, although the distance information Z used at the time of 
coordinate transformation cannot be found in any fields other than the 
overlap section since corresponding points are not extracted, the distance 
information on a background is used instead. 

[0070] An example 2, next the 2nd example of this invention are explained. 
Drawing 6 is drawing showing the configuration of the image synthesizer 
unit of the 2nd example of this invention. Left-hand side image pick-up 
system 610L in drawing 6 , right-hand side image pick-up system 610R, 
image memories 620L and 620R, The parameter storage section 621, the 
address generation section 622, the corresponding-points extract section 623, 
the distance information calculation section 624, the coordinate 
transformation section 625, the image data-interpolation sections 626R and 
626L, Each of the image composition section 627 and the image memory 628 
for an output Left-hand side image pick-up system 510L shown in drawing 5 , 
right-hand side image pick-up system 510R, image memories 520L and 520R, 
Since it is what performs the same actuation as the parameter storage 
section 521, the address generation section 522, the corresponding-points 
extract section 523, the distance information calculation section 524, the 
coordinate transformation section 525, the image data-interpolation sections 
526R and 526L, the image composition section 527, and the image memory 
528 for an output, explanation is omitted. 

[0071] Unlike the 1st example which the point of using the output of the 
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image correlation section 630 as decision of the field of the image 
composition section 627 showed to drawing 5 , this example explains this 
portion hereafter. 

[0072] The output of the image data-interpolation sections 626L and 626R is 
once made into image memories 629L and 629R account 100 million, 
respectively. And it is judged as the field where it is judged as what has the 
contents of two image memories the same [ the value of two image memories 
is cut off by the small field above, and the similarity of each smallness field is 
called for in the image correlation section 630 by correlation count, and are 
compared, and ] when higher than a value predetermined in a correlation 
value, and a long distance portion is hidden with a short- distance body when 
lower than a predetermined value. 

[0073] In the image composition section 627, only when the contents of two 
image memories are judged to be the same thing, the average value of the 
interpolation image data carried out account 100 million is outputted to 
image memories 629L and 629R as synthetic image data as processing of the 
overlap section. The processing in other than the above is the same as 
processing of the image composition section of the 1st example, or [ that the 
interpolation image data memorized by which of image memories 629L and 
629R is outputted as synthetic image data at this time ] - image memories 
629L and 629R ■■ correlation with the image of a smallness field will be 
taken in each interior soon, and the one where a correlation value is more 
higher is chosen noting that the background image is interpolated smoothly. 
The method of choosing a interpolation image using correlation by this 
example can offer an image with few image distortions as an output image, 
also when corresponding-points extract processing is not performed 
completely. 

[0074] Moreover, the method using the result of the corresponding-points 
extract section 623 shown in the 1st example with the output of the image 
correlation section 630 may be combined as decision of the field of the image 
composition section 627 in the image correlation section 630. In this case, 
selection of a interpolation image becomes more certain and the portion into 
which the amount of [ a body portion and ] background laps, and image 
quality deteriorates decreases more. 
[0075] 

[Effect of the Invention] Since this invention is constituted as explained 



22/25 



Japanese Publication number : 08-018857 



above, an effect which is indicated below is done so. 

[0076] In the method and the thing according to claim 3 of claim 1, also when 
it has the distance distribution whose photographic subject contains the body 
with which distance differs remarkably to a background, a location gap of an 
image arises, or it is effective in the amount of [ a body portion and ] 
background not lapping, and being able to obtain few synthetic images of 
deterioration of image quality. 

[0077] In addition to each above-mentioned effect, in a method and a thing 
according to claim 3 according to claim 2, there is an effect which can 
compound a more nearly actually near exact image. 

[0078] In a thing according to claim 5, even if it is the case where 
corresponding-points extract processing is not performed completely in 
addition to each above-mentioned effect, there is an effect which can create 
few synthetic images of distortion with which smooth interpolation 
processing was performed. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

fDrawing l] It is drawing showing basic arrangement of the image pick-up 
system in one example of the compound eye image pick-up equipment of this 
invention. 

[Drawing 2] It is the outline block diagram showing the configuration of the 
left-hand side image pick-up optical system shown in drawing 1 , and 
right-hand side image pick-up optical system. 

[Drawing 3] It is the block diagram showing the configuration of the 
angle -of- convergence control system in one example of the compound eye 
image pick-up equipment of this invention. 

[Drawing 4] It is drawing for explaining the formation image after synthetic 
processing. 

fDrawing 5] It is the block diagram showing the configuration of the image 
convolution transformation processing section in the 1st example of this 
invention. 

[Drawing 6] It is the block diagram showing the configuration of the image 
convolution transformation processing section in the 2nd example of this 
invention. 
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[Drawing 71 It is a conceptual diagram to show an example of the compound 
eye image pick-up equipment which a part of **** is made to overlap, 
compounds two images which picturized the common photographic subject 
and were obtained using two image pick-up systems, and creates one image 
of the aspect ratio of arbitration. 

[Drawing 8] It is drawing showing basic arrangement of a compound eye 
image pick-up system. 

[Drawing 9] It is drawing for explaining the creation method of the 
control-objectives value of an angle of convergence. 

[Drawing 10] It is drawing for explaining the formation image after synthetic 
processing. 

[Description of Notations] 

1 Photographic Subject Plane 

2L Left-hand side body side 

2R Right-hand side body side 

10L Left-hand side image pick-up system 

10R Right-hand side image pick-up system 

11L Left-hand side image pick-up optical system 

11R Right-hand side image pick-up optical system 

12L Left-hand side image sensors 

12R Right-hand side image sensors 

21L, 21R, 23L, 23R Lens group 

22L, 22R Variable power group 

24L, 24R Focus group 

25L, 25R Zoom motor 

26L, 26R Focal motor 

27L, 27R Angle -of- converge nee motor 

28L, 28R Angle-of-convergence encoder 

41L, 41R Zoom encoder 

42L, 42R Focal encoder 

50 Angle-of-Convergence Control System 

51 Operation Control Section 

52 Angle-of-Convergence Control Section 
60 System Controller 

510L, 610L Left-hand side image pick-up system 
510R, 610R Right-hand side image pick-up system 
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520L, 520R, 620L, 620R, 629L, 629R Image memory 

521.621 Parameter storage section 

522.622 Address generation section 

523.623 Corresponding-points extract section 

524.624 Distance information calculation section 

525.625 Coordinate transformation section 

526L, 526R, 626L, 626R Image data-interpolation section 

527.627 Image composition section 

528.628 Image memory for an output 
630 Image Correlation Section 

LL, LR Optical axis 
theta Angle 

2theta Angle of convergence 

OL ~ OR - CL - CR - AL - BL - CL - DL - AR - BR CR - DR - AL - ' - 

BL - ' - CL - ' - DL - ' - AR - ' - BR - ' - CR - • - DR'EL - floor line - ER 

- FR"E-- ? --F--'" Point 

IL, IR Image surface 

IL+R Virtual image surface 

H, H\ V, V 1 , and V - " - Length 

x y Shaft 
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